A Bayesian Kernel Logistic Discriminant Model: An Improvement to the Kernel Fisher's Discriminant

The Kernel Fisher’s Discriminant (KFD) is a non-linear classifier which has proven to be powerful and competitive to several state-of-the-art classifiers. Its main ingredient is the kernel trick which allows the efficient computation of Fisher’s Linear Discriminant in feature space. However, it is assuming equal covariance structure for all transformed classes, which is not true in many applications. In this paper, we propose a novel Bayesian Kernel Logistic Discriminant model (BKLD) which goes one step further by representing each transformed class by its own covariance matrix. This can allow more flexibility and better classification performances than the KFD. The posterior distribution of the BKLD model is elegantly approximated by a tractable Gaussian form using variational transformation and Jensen’s inequality, which allow a straightforward computation of the weights. An extensive comparison of the BKLD to the KFD and to other state-of-the-art non-linear classifiers is performed. Also, analysis of algorithm complexity and numerical accuracy is provided.